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Already not so big :) 
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e Big Fucking Routers: 
o Pros: 


m Included hardware HA (NxPSU, NxRE) 

Full Internet table in FIB 

Support of various routing protocols (e.g. IS-IS) 
Support of long range optics (port power out) 
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e Big Fucking Routers: 
o Cons: 


Too expensive (and sometimes you must buy a 
license) 

Too complicated 

A lot of power consumption 

A lot of rack space 

Not flexible software 
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e Switches: 
o Pros: 


m Not so expensive 

Less power consumption (not always) 

Less rack space (not always) 

A lot of 100G/400G ports per RU 

BYOS (Bring your own software, in Linux case) 
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e Switches: 
o Cons: 


m FIB limits (you cannot instal BGP FV in FIB) 
Fixed hardware configuration (not always) 
Not reserved RE (not always) 
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e Switches types: 
o By hardware: 
m Own chips (Cisco Monticello, Juniper Paradise(Q5)) 
m Commodity chips (Broadcom, Intel, Cavium, 
Barefoot) 
o By software: 
m Proprietary OS (IOS, Junos, EOS) 
m Whitebox (You can change OS) 


Normalized - % Activelln-Use BW per Chipset Family 
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Credits Jonah Gustawsson: https://twitter.com/Gustawsson/status/1353029596661354496 12 


L JARATORLABS == Whitebox $ witches: 
mm 
e Software 
o MLNX-OS/Mellanox Onyx (DNYX 
Cumulus 
SDK 
SAI (Switch Abstraction Interface), SONIC (NOS) SJ SONIC 
switchdev (Linux kernel) 


e https://www.mellanox.com/products/switch-software 


O O © © 


L JaRATOR LABS Switch dev- 


(L2) | Crate onto) (>) | User Space 
e in-kernel infrastructure 
e dataplane — Linux (offload) 


Notifications 


o bridging mixsw. spectrum 
o routing et ovna s. aran Kernel 
o filtering | zie | 
e since 2014 sca 
e Mellanox (2015) | 
Spectrum system 


pN 
Courtesv of Mellanox Technologies 
https://blog.mellanox.com/2018/12/mellanox-spectrum-linux-switch-powered-bv-switchdev/ 
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e kernel version 
o vanilla (https://github.com/Mellanox/mixsw/wikifimixsw) 
o net-next 
e firmware 
o in driver (linux > 4.13, fw = 13.1420.122) 
o tool (mstflint) 
e initramfs 
o premature driver load 
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e iproute2 

o Ip 

o bridge 

o devlink 

o tc 

ethtool 

Ildpad: LLDP, QoS (DCB) 
e sysctl: hash policy, gos prio update 
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LACP (link aggregation) 
VLAN, bridge (switching) 

VRF (virtual routers) 

ECMP (multipath) 

ACL (filtering) 

GRE (tunneling) 

no PBR (policy based routing) 


EEE „Configuration way differ ces: 


e Monolith configuration vs different configurations files 
o Traditional NOS uses monolith configuration: 
m One big configuration divided by blocks (RP, ACL, 
System services, Interfaces configuration). 
m “Syntax sugar” 
o In Linux we have many different config files: 
m You must keep in mind file locations and/or change 
order 
m sed, awk, grep, ect... 


Sito -Configuration way differences. 


il Pari ie 
dl l | 
| 

(il li 7 Ve 


ye C 


CORMOREABS __ Cisco/Juniper LACP exam le. 
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ASS Cisco/Juniper vlan example. 
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CORMOREABS Cisco/Juniper VRF example: 
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Linux VRF example 


netns / vrf 


special ip rule (V4, ve) 


add vrf: link type vrf, vrf — table 


ip link add name vrf-int type vrf table 200 


iface to vrf: ip link set master 


ip link set dev vlan20 master vrf-int 


route between vrfs: explicit dev 


ip route add 203.0.113.0/24 via 198.51.100.2 dev vlan20 table 100 
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CORMOREABS Cisco/Juniper ACL example. 
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port — bond — bridge — vlan, loopback — vrf — ip 
restrictions 

o down before set master (port, bond) 

o can not set master to enslaved (bond, bridge) 
init: big script 

runtime changes 
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ond) 
o cannot set master to kond, bridge) 
init: big script JA 
runtime changes 
o pain 


29 


e perltakes care 
o mixrtr 


split 4 

slave port1/0, port1/1 
slave port1/2, port1/3 
native port2 

vrf ext 

ip 192.0.2.2/31 

tag bond srvl, bond srv2 


vrf int 
ip 198.51.100.1/24 


mixrtr 


vrf ext 
ip 192.0.2.1/32 


table 
route 


table 
route 
route 


100 
0.0.0.0/0 via 198.51.100.2 dev vlan20 


200 
0.0.0.0/0 via 192.0.2.3 dev vlan10 
203.0.113.0/24 via 198.51.100.2 dev vlan20 


Config example (init, split; link) 


L JaRATOR LABS 


sysctl -w … 

ip rule del pref 0 

ip rule add pref 30000 table local 

devlink port split pci/0000:01:00.0/25 count 4 

tc gdisc add dev enpls0npls0 ingress block 100 ingress 


ip link add name bond srvl type bond lacp rate fast min links 1 \ 
mode 802.3ad xmit hash policy layer3+4 
ip link set dev bond srvl down 


ip link add name loop10 type dummy 

ip link set dev loopl0 down 

ip link add name switch type bridge vlan filtering 1 
ip link set dev switch down 

ip link add name vrf-ext type vrf table 100 

ip link set dev vrf-ext down 

ip link add name vrf-int type vrf table 200 

ip link set dev vrf-int down 


Config example (set masters) 
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dev enpls0npls0 down 
dev enpls0npls0 master bond srvl 
dev enpls0npls0 down 


dev enplsOnp2 master switch 
dev enplsOnp2 down 
dev bond srvl master switch 
dev bond srvl down 


dev loop10 master vrf-ext 

dev loopl0 down 

link switch name vlan10 type vlan id 10 
dev vlanl0 down 


dev vlanl0 master vrf-ext 
dev vlan10 down 
dev vlan20 master vrf-int 
dev vlan20 down 


Config example (vlan, link up) 
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bridge vlan vid 1 dev bond srvl 
bridge vlan vid 20 dev bond srvl 


bridge vlan vid 10 dev enpls0np2 pvid untagged 
bridge vlan vid 10 dev switch self 

bridge vlan vid 20 dev switch self 

ip link set enplsOnpis0 up 


ip link set bond srvl up 
ip link set bond srv2 up 
ip link set loopl0 up 
ip link set switch up 
ip link set vlanl0 up 
ip link set vlan20 up 
ip link set vrf-ext up 
ip link set vrf-int up 


Config example (ip; route) 
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address add 192.0.2.1/32 dev loop10 
address add 192.0.2.2/31 dev vlanl0 
address add 198.51.100.1/24 dev vlan20 


route replace 0.0.0.0/0 metric 0 table 100 proto static X 
nexthop via 198.51.100.2 dev vlan20 weight 1 

route replace blackhole 0.0.0.0/0 metric 4278198272 N 
table 100 proto static 


route replace 0.0.0.0/0 metric 0 table 200 proto static X 
nexthop via 192.0.2.3 dev vlanl0 weight 1 

route replace blackhole 0.0.0.0/0 metric 4278198272 N 
table 200 proto static 

route replace 203.0.113.0/24 metric 0 table 200 X 
proto static nexthop via 198.51.100.2 dev vlan20 weight 1 


e moveporttoother bond 


ip 
ip 
ip 
ip 
ip 
ip 


ip 
ip 
bridge vlan 
bridge vlan 
ip link set 
ip link set 


link 
link 
link 
link 
link 
link 
link 
link 


set 
set 
set 
set 
set 
set 
set 
set 


enpls0npls2 down 
enpls0npls2 nomaster 
bond srvl down 

bond srvl nomaster 
enplsOnpls2 master bond srvl 
enpls0npls2 down 

bond srvl master switch 
bond srvl down 

vid 1 dev bond srvl 

vid 20 dev bond srvl 
enpls0npls2 up 

bond srvl up 


Change example 
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tc (qdisc, filter) 

e routed 8 bridged 

e shared ac 
o block (newer tc) 

e per-portoniv 
goto 
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ACL 


tc (qdisc, filter) 
routed & bridged 
shared ac 
o block (newer tc) 
per-port only 


ip proto icmp dst ip 192.0.2.2 action pass 


goto src ip 203.0.113.0/24 action drop 
dst ip 203.0.113.0/24 action goto [ex1] 
mixacl dst ip 203.0.113.0/24 action drop 


action pass 


o chain per vlan 


ip proto icmp action pass 


o chain 0: match vlan FESSES ESS EEE 


action drop 
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tc filter add 


. protocol 
. protocol 
. protocol 


. protocol 
. protocol 
. protocol 
. protocol 
. protocol 


. protocol 
. protocol 


ip 
ip 
ip 


ip 
ip 
ip 
ip 
ip 


block 


chain 
chain 
chain 


chain 
chain 
chain 
chain 
chain 


100 … 


101 
101 
101 


100 
100 
100 
100 
100 


802.1q chain 
802.1q chain 


flower 
flower 
flower 


flower 
flower 
flower 
flower 
flower 


ACL example 


ip proto icmp action pass 
ip proto tcp action pass 
action drop 


ip proto icmp dst ip 192.0.2.2 action pass 
src ip 203.0.113.0/24 action drop 

dst ip 203.0.113.0/24 action goto chain 101 
dst ip 203.0.113.0/24 action drop 

action pass 


1 flower vlan id 10 action goto chain 100 
2 flower action pass 
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tc filter add 
protocol ip 
tc filter add 
protocol ip 
tc filter add 
protocol ip 
tc filter add 
protocol ip 


tc filter add 


block 
chain 
block 
chain 
block 
chain 
block 
chain 


block 


100 
102 
100 
102 
100 
102 
100 
102 


100 


protocol 802.1q chain 
tc filter add block 100 
protocol 802.1q chain 


tc filter del block 100 
tc filter del block 100 
tc filter del block 100 


: apply changes 


N 
pref 1 flower src ip 203.0.113.0/24 action drop 


N 
pref 2 flower dst ip 203.0.113.0/24 action goto chain 100 


N 
pref 3 flower dst ip 203.0.113.0/24 action drop 


X 


pref 4 flower action pass 


X 
0 pref 3 flower vlan id 10 action goto chain 102 
N 


O pref 4 flower action pass 


chain 0 pref 1 
chain 0 pref 2 
chain 101 
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filter protocol ip pref 1 flower chain 
filter protocol ip pref 1 flower chain 

eth type ipv4 

ip proto icmp 

in hw 

action order 1: gact action pass 
random type none pass val 0 
index 1 ref 1 bind 1 


filter protocol ip pref 2 flower chain 
filter protocol ip pref 2 flower chain 

eth type ipv4 

ip proto tcp 

in hw 

action order 1: gact action pass 
random type none pass val 0 
index 2 ref 1 bind 1 


filter protocol ip pref 3 flower chain 
filter protocol ip pref 3 flower chain 
eth type ipv4 
in hw 
action order 1: gact action drop 


100 
100 handle 0x1 


100 
100 handle 0x1 


100 
100 handle 0x1 


ACL tc filter show 
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filter protocol ip pref 1 flower chain 
filter protocol ip pref 1 flower chain 

eth type ipv4 

ip proto icmp 

in hw 

action order 1: gact action pass 
random type none pass val 0 
index 1 ref 1 bind 1 


filter protocol ip pref 2 flower chain 
filter protocol ip pref 2 flower chain 

eth type ipv4 

ip proto tcp 

in hw 

action order 1: gact action pass 
random type none pass val 0 
index 2 ref 1 bind 1 


filter protocol ip pref 3 flower chain 
filter protocol ip pref 3 flower chain 
eth type ipv4 
in hw 
action order 1: gact action drop 


100 
100 handle 0x1 


100 
100 handle 0x1 


100 
100 handle 0x1 


ACL tc filter show 
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https://gitlab.com/gratorlabs/mlxtoolkit 
MIT license 

Perl 

mixacl: 1k lines 

mlxrtr: 2.7k lines 

dependencies: 

o perl modules 

o /root/bin/{bridge,ip,tc} 

o devlink, syscti 
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https://gitlab.com/gratorlabs/mlxtoolkit 
MIT license 

Perl 

mixacl: 1k lines 

mixrtr: 2./k lines 
dependencies: 

o perl modules 

o /root/bin/{bridge,ip,tc} 
o devlink, syscti 
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e ESC, R, ESC, r, ESC, R 
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o ESC, R, ESC, r, ESC, R 
o BIOS "Ctrl-Alt-Del” 
e SysRa: use “break” 
o minicom: Ctrl+a, Ctrl-f 
o screen: Ctrl+a, Ctrl+b 
e BIOS: Ctrl-b 
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e My contacts 
o Dmitry Shemonaev 
o ds@arator.net 


